Towards a declarative, reproducible, homogeneous,
cross-platform command-line environment across
remote HPC machines (with live demonstration pretending to be a masterhacker!) The STEP-UP RSLondon Conference 2025 Dr Krishnakumar Gopalakrishnan, University College London July 07, 2025 Who am I? • Research Software Engineer at UCL ARC
• Rides bikes on weekends • Uses Arch btw • Test cricket fan 2 / 38 Who am I? • Research Software Engineer at UCL ARC
• Rides bikes on weekends • Uses Arch btw • Test cricket fan • Command-line enthusiast # So in the ongoing Anderson-Tendulkar Trophy
——————————————————— [finished] ———————————————————
3 / 38 The Unix Philosophy ▍ This is the Unix philosophy:
▍ Write programs that do one thing and do it well.
▍ Write programs to work together.
▍ Write programs to handle text streams, because that is a universal interface.
Doug McIlroy, Bell Labs Computing Sciences Research Center https://en.wikipedia.org/wiki/Unix_philosophy#Do_One_Thing_and_Do_It_Well 4 / 38 Command-line interfaces and HPC systems • The default user interface on HPC systems • Users are provided with Secure Shell (SSH) access • Edit/Compile/Debug cycle on remote systems • Job submission workflow and non-interactive compute node use that uses schedulers 5 / 38 Command-line interfaces and HPC systems • The default user interface on HPC systems • Users are provided with Secure Shell (SSH) access • Edit/Compile/Debug cycle on remote systems • Job submission workflow and non-interactive compute node use that uses schedulers ▍ Note
▍
▍ Often not emphasised: interactive use e.g. text editing, searching, command history ...
6 / 38 Towards a homogeneous environment ▍ Goal
▍
▍ Need an environment identical to personal workstation on all remote hosts (HPCs, VMs, Containers ...)
██ Workstation platforms (OS/CPU Architectures) Operating System │ x86 (32-bit) │ x86_64 (64-bit) │ ARMv7 (32-bit) │ ARM64 (AArch64) │ RISC-V │ PowerPC ─────────────────┼──────────────┼─────────────────┼────────────────┼─────────────────┼────────┼──────── Windows │ │ │ │ │ │ Linux │ │ │ │ │ │ *BSD │ │ │ │ │ │ macOS │ │ │ │ │ │ AIX │ │ │ │ │ │ Haiku │ │ │ │ │ │ ReactOS │ │ │ │ │ │ RedoxOS │ │ │ │ │ │
7 / 38 ██ Realistic support matrix for combined Desktop/HPC Operating System │ x86_64 (64-bit) │ ARM64 (AArch64) ─────────────────┼─────────────────┼──────────────── Linux │ ✅ │ ✅ macOS │ ❓ │ ✅ Windows │ ❓ │ ❓ ██ Common environment? • Shell startup files (aliases, environment variables, cleanup ...) • Cross-platform CLI/TUI tools • Configuration files for tooling 8 / 38 Hands-on example 9 / 38 Scenario ██ All hosts in the inventory 10 / 38 Consider only remote (HPC) hosts in the inventory hostname
——————————————————— [finished] ———————————————————
blend 11 / 38 ██ Remote hosts in the inventory 12 / 38 Config file synchronisation across hosts • Clone/sync on remote host with a Version Control System (VCS) • A dotfiles manager (https://dotfiles.github.io/)
◦ GNU Stow ◦ dotbot ◦ rcm ◦ and many many more! 13 / 38 Chezmoi • I have settled on using Chezmoi (pronounced shay-mwa)
• Full power of go templates # chezmoi data
——————————————————— [finished] ———————————————————
14 / 38 Chezmoi templating $HOME/.config/git/config
[user]
{{- if eq chezmoi.hostname "blend" }}
email = "personal_email@personal_host.co.uk"
{{- else }}
email = "work_email@work.ac.uk"
{{- end }}
user@blend $ chezmoi apply
[user]
email = "personal_email@personal_host.co.uk" 15 / 38
Deploy on all systems ██ Use a configuration management tool • Ansible (agentless, push model) • Puppet • Chef • Salt stack • Pyinfra
16 / 38 Tooling/Packages 17 / 38 Combinatorial explosion of packages ▍ Consider
▍ m: different processor architectures (target CPUs),
▍ n: different operating systems,
▍ p: different executable tools.
Total number of binaries to be managed: Even with 3 CPU architectures and 20 tools to be managed, we have about 60 executables to manage, from which the valid binaries for each platform must be identified and deployed on the p-th machine. 18 / 38 Managing package lifecycle ▓▓▓ Install (?), Update to latest, Pin versions of selected packages/tools, Remove packages 19 / 38 Managing package lifecycle ▓▓▓ Install (?), Update to latest, Pin versions of selected packages/tools, Remove packages act cargo:tlrc fx jinja-lsp
actionlint ccache fzf jless
age checkmake gdu jnv
asdf:richin13/asdf chezmoi gh jq
-neovim clang-tools git k9s
awk-language-serve cmake git-cliff kubectl
r cosign gitui lazydocker
bat cppcheck glow lazygit
bat-extras croc go lazyjournal
beautysh curl go:crictty less
bingrep delta go:daylight ltex-ls
binsider difftastic go:kubecolor lua
biome direnv go:stew lua-language-serve
bottom diskonaut goawk r 20 / 38
broot dive graphviz lychee
Linters, Formatters, Language Runtimes, Compilation frameworks, Utilities ... minikube pipx procs usage
nasm pipx:ansible-d pylyzer uv
neocmakelsp ev-tools python vault
ninja pipx:ansible-i ripgrep vlang
node nventory-graph ripgrep-all viddy
npm:ansible-langua er ruff xz
ge-server pipx:ansible-l rust yamlfmt
npm:mermaid-cli int selene yamllint
npm:compose-langua pipx:argcomple shellcheck yj
ge-service te shellharden zig
npm:alex pipx:bandit shfmt zk
npm:bash-language- pipx:basedpyri starship
server ght stylua
npm:bibtex-tidy pipx:clang-for taplo 21 / 38
npm:commitlint mat terraform # Let's count the
How to manage these software tools/packages everywhere? 22 / 38 ██ System package manager? • apt-get/apt • yum/dnf • zypper • pacman ▓▓▓ Let's try which vim
————————————— [finished with error] ——————————————
which: no vim in (/home/chezmoi_trial1/.config/pixi/condabin:/home/chezmoi_trial1/.local/bin:/home/ chezmoi_trial1/.local/share/mise/installs/act/0.2.79:/home/chezmoi_trial1/.local/s hare/mise/installs/actionlint/1.7.7:/home/chezmoi_trial1/.local/share/mise/install s/age/1.2.1/age:/home/chezmoi_trial1/.local/share/mise/installs/bat/0.25.0:/home/c
hezmoi_trial1/.local/share/mise/installs/bat-extras/2024.08.24/bin:/home/chezmoi_t 23 / 38
rial1/.local/share/mise/installs/biome/2.0.6:/home/chezmoi_trial1/.local/share/mis Let us install vim # which OS are we on?
——————————————————— [finished] ———————————————————
NAME="blendOS" PRETTY_NAME="blendOS" ID=blendos BUILD_ID=rolling ANSI_COLOR="38;2;23;147;209" HOME_URL="https://blendos.co/" DOCUMENTATION_URL="https://docs.blendos.co/" SUPPORT_URL="https://github.com/blend-os" BUG_REPORT_URL="https://github.com/blend-os"
LOGO=blendos-logo 24 / 38
pacman -S vim
————————————— [finished with error] ——————————————
error: you cannot perform this operation unless you are root. 25 / 38 ▓▓▓ Cannot do this on remote HPC systems. But let us on our local machine su krishnakumar
sudo pacman -S vim
26 / 38 Software Package Management on a HPC system • Curated list of scientific software pre-installed (and upon request) • Versions are user-switchable by a modules system (environment-modules/Lmod) • Custom user-modules can be loaded • Unpriveleged container execution with container runtimes like Apptainer/Singularity/Podman? 27 / 38 Software Package Management on a HPC system • Curated list of scientific software pre-installed (and upon request) • Versions are user-switchable by a modules system (environment-modules/Lmod) • Custom user-modules can be loaded • Unpriveleged container execution with container runtimes like Apptainer/Singularity/Podman? ▍ 🤨
▍
▍ But we are talking about installing simple productivity tools and keeping them up-to-date
▍ Not scientific libraries and programs
28 / 38 Popular HPC packaging ecosystem 29 / 38 • Intended for scientific software and libraries (GROMACS, LAMMPS, MPI ...) • Support complex build provenances (MPI libraries, dependency versions, compilation toolchain ...) • Latest versions of utility tools often not available 30 / 38 So, what are the options available? 31 / 38 Nix was it for me! 32 / 38 However! 33 / 38 So, is there a solution? ██ Desired characteristics • Declarative/Idempotent • Reproducible • Installable/Runnable from user home directory • Cross-platform 34 / 38 ██ pixi global • install • edit • sync • manifest file pixi-global.toml
35 / 38 ██ Mise ██ pixi global • install • use -g
• edit • prune
• sync • global config.toml
• manifest file pixi-global.toml
36 / 38 Keeping tools up to date 37 / 38 Further resources ▓▓▓ Worflow configuration codes, presentation in HTML & PDF formats 38 / 38